A Grammatico-Statistical Approach To Discourse Partitioning

I n t r o d u c t i o n In this paper, we describe a method for discovering coherent texts from the uns t ruc tured corpus. T h e method is bo th linguistically and statistically motivated. It derives a linguistic motivat ion front the view tha t discourse consists of what we call discourse segments , minimal coherent mtits of discourse (Nomoto and Nitta, 1993), while statist ically it is guided by ideas from intbrmatiou retrieval (Salton, 1988). Previous quant i ta t ive approaches to text segmentat ion (Hearst, 1993; Kozima, 1993; Youmans, 1991) have paid little a t ten t ion to a statistically impor tan t s t ructure tha t a discourse might have and detined it away ~Ls a lump of words or sentences. 1'art of our concern here is with explicating possible effects of a discourse segment on the quant i ta t ive s t ructur ing of discourse. In what follows, we will describe some impor tan t features about discourse segment and see how it can be incorporated into a statist ical analysis of discourse. Also some comparison is made with other apl)roaches such as (Youmans, 1991), followed by discussion on the results of the present method. Theory of D i s c o u r s e S e g m e n t T h e theory o f discourse seF, m e n t (Nomoto a.nd Nitta, *2520 | t a t o y a m a S a i t a m a 350-03 J a p a n tel. +81-492-9C~6111 fax. +81-492-9(~-6006 1993) carries with it a set of empirical hypotheses about s t ructure of Japanese discourse. Among them is the claim tha t Japanese discourse is constructed from a series of linguistics units called discourse segment . T h e discourse segment is thought of az a topic-comment structure, where a topic corresponds to the subject mat te r and a comment a discussion about it. In particular, Japanese haz a special way of marking the topic: by sutllxing it with a postposit ionM particle wa. Thus in Japanese, a topic-comment s t ruc ture takes the form: